Comparison of effect estimates between preprints and peer-reviewed journal articles of COVID-19 trials.

BACKGROUND: Preprints are increasingly used to disseminate research results, providing multiple sources of information for the same study. We assessed the consistency in effect estimates between preprint and subsequent journal article of COVID-19 randomized controlled trials. METHODS: The study utilized data from the COVID-NMA living systematic review of pharmacological treatments for COVID-19 (covid-nma.com) up to July 20, 2022. We identified randomized controlled trials (RCTs) evaluating pharmacological treatments vs. standard of care/placebo for patients with COVID-19 that were originally posted as preprints and subsequently published as journal articles. Trials that did not report the same analysis in both documents were excluded. Data were extracted independently by pairs of researchers with consensus to resolve disagreements. Effect estimates extracted from the first preprint were compared to effect estimates from the journal article. RESULTS: The search identified 135 RCTs originally posted as a preprint and subsequently published as a journal article. We excluded 26 RCTs that did not meet the eligibility criteria, of which 13 RCTs reported an interim analysis in the preprint and a final analysis in the journal article. Overall, 109 preprint-article RCTs were included in the analysis. The median (interquartile range) delay between preprint and journal article was 121 (73-187) days, the median sample size was 150 (71-464) participants, 76% of RCTs had been prospectively registered, 60% received industry or mixed funding, 72% were multicentric trials. The overall risk of bias was rated as 'some concern' for 80% of RCTs. We found that 81 preprint-article pairs of RCTs were consistent for all outcomes reported. There were nine RCTs with at least one outcome with a discrepancy in the number of participants with outcome events or the number of participants analyzed, which yielded a minor change in the estimate of the effect. Furthermore, six RCTs had at least one outcome missing in the journal article and 14 RCTs had at least one outcome added in the journal article compared to the preprint. There was a change in the direction of effect in one RCT. No changes in statistical significance or conclusions were found. CONCLUSIONS: Effect estimates were generally consistent between COVID-19 preprints and subsequent journal articles. The main results and interpretation did not change in any trial. Nevertheless, some outcomes were added and deleted in some journal articles.

What are preprints?

Preprints are a way in which a manuscript containing scientific results can be rapidly communicated from one scientist, or a a group of scientists, to the entire scientific community. This video by ASAPbio (Accelerating Science and Publication in biology) explains what preprints are and their benefits, how they differ from journal publications, and how scientists can use both mechanisms to communicate their work

Examining linguistic shifts between preprints and publications.

Preprints allow researchers to make their findings available to the scientific community before they have undergone peer review. Studies on preprints within bioRxiv have been largely focused on article metadata and how often these preprints are downloaded, cited, published, and discussed online. A missing element that has yet to be examined is the language contained within the bioRxiv preprint repository. We sought to compare and contrast linguistic features within bioRxiv preprints to published biomedical text as a whole as this is an excellent opportunity to examine how peer review changes these documents. The most prevalent features that changed appear to be associated with typesetting and mentions of supporting information sections or additional files. In addition to text comparison, we created document embeddings derived from a preprint-trained word2vec model. We found that these embeddings are able to parse out different scientific approaches and concepts, link unannotated preprint-peer-reviewed article pairs, and identify journals that publish linguistically similar papers to a given preprint. We also used these embeddings to examine factors associated with the time elapsed between the posting of a first preprint and the appearance of a peer-reviewed publication. We found that preprints with more versions posted and more textual changes took longer to publish. Lastly, we constructed a web application (https://greenelab.github.io/preprint-similarity-search/) that allows users to identify which journals and articles that are most linguistically similar to a bioRxiv or medRxiv preprint as well as observe where the preprint would be positioned within a published article landscape.

Curating the Evidence About COVID-19 for Frontline Public Health and Clinical Care: The Novel Coronavirus Research Compendium.

The public health crisis created by the COVID-19 pandemic has spurred a deluge of scientific research aimed at informing the public health and medical response to the pandemic. However, early in the pandemic, those working in frontline public health and clinical care had insufficient time to parse the rapidly evolving evidence and use it for decision-making. Academics in public health and medicine were well-placed to translate the evidence for use by frontline clinicians and public health practitioners. The Novel Coronavirus Research Compendium (NCRC), a group of >60 faculty and trainees across the United States, formed in March 2020 with the goal to quickly triage and review the large volume of preprints and peer-reviewed publications on SARS-CoV-2 and COVID-19 and summarize the most important, novel evidence to inform pandemic response. From April 6 through December 31, 2020, NCRC teams screened 54 192 peer-reviewed articles and preprints, of which 527 were selected for review and uploaded to the NCRC website for public consumption. Most articles were peer-reviewed publications (n = 395, 75.0%), published in 102 journals; 25.1% (n = 132) of articles reviewed were preprints. The NCRC is a successful model of how academics translate scientific knowledge for practitioners and help build capacity for this work among students. This approach could be used for health problems beyond COVID-19, but the effort is resource intensive and may not be sustainable in the long term.

Preprint servers in lipidology: current status and future role.

PURPOSE OF REVIEW: Preprinting, or the sharing of non-peer reviewed, unpublished scholarly manuscripts, has exploded in all fields of science and medicine over the past 5 years. We searched the literature and evaluated the posting and uptake of preprint publications in the field of lipidology in bioRxiv and medRxiv servers. We also contacted the editorial offices of 20 journals that publish original research in lipidology to gauge their policies on preprints. RECENT FINDINGS: All 20 journals contacted indicated that they accepted preprints. As of 31 May 2021, 473 and 231 preprints in lipidology had been submitted to bioRxiv and medRxiv, respectively. About half of all lipidology preprints were related to cardiovascular, cardiometabolic, and/or metabolic diseases (CVMD) and their risk factors, but at least 12 other disease categories were also represented. 16.9% and 1.08% of medRxiv and bioRxiv preprints, respectively, were related to coronavirus disease 2019 (COVID-19). SUMMARY: All identified journals accept lipidology themed preprints for submission, removing any barriers authors may have had regarding preprinting. Based on growing experience with preprinting, this trend should encourage increased community feedback and facilitate higher quality lipidology research in the future.

Woman authorship in pre-print versus peer-reviewed oral health-related publications: A two-year observational study.

OBJECTIVES: Women in oral health science face similar societal issues and challenges as those in other STEMM careers, and gender disparities continue to exist as evidenced by fewer women represented as first and last authors in scientific publications. Pre-prints may serve as a conduit to immediately disseminating one's work, bypassing the arduous peer review process and its associated inherent biases. Therefore, the purpose of this study was to 1] compare the gender of first and last authors in pre-print versus peer reviewed publications, 2] examine the composition of first and last author pairs as stratified by publication type, and 3] examine the correlation between woman authorship and institutional geographic location and publication metrics stratified by publication type. METHODS: The keyword "oral health" was used to search for publications in BioRxiv and Pubmed in the years 2018 and 2019. Gender of first and last authors were determined, and its frequency was considered as the primary outcome. Additionally, the geographic location of the author's associated institution and publication metrics measured by Altmetrics score were extracted. Data was descriptively summarized by frequencies and percentages. Chi-square analysis was conducted for categorical variables which included the relationship between gender and publication type as well as gender and region of author's associated institution. Binomial regression analysis was conducted to analyze the relationship between gender and Altmetrics. RESULTS: Woman first authors comprised 40.3% of pre-prints and 64.5% of peer reviewed publications [p<0.05]. Woman last authors comprised 31.3% of pre-prints and 61.5% of peer reviewed publications [p<0.05]. When analyzing the relationships between first and last author, the Man-Man pairing represented 47.7% of the pre-print publications and the Woman-Woman pairing comprised a majority of the of the peer review publications at 47.5%. All results were statistically significant with a p-value <0.05. No significant correlation was found between region of institution or Altmetrics and gender of first or last authors [p>0.05]. CONCLUSION: For the first time in oral health science, it was found that women show higher representation as first and last author positions in peer reviewed publications versus pre-prints.

Preprint articles as a tool for teaching data analysis and scientific communication.

The skill of analyzing and interpreting research data is central to the scientific process, yet it is one of the hardest skills for students to master. While instructors can coach students through the analysis of data that they have either generated themselves or obtained from published articles, the burgeoning availability of preprint articles provides a new potential pedagogical tool. We developed a new method in which students use a cognitive apprenticeship model to uncover how experts analyzed a paper and compare the professional's cognitive approach to their own. Specifically, students first critique research data themselves and then identify changes between the preprint and final versions of the paper that were likely the results of peer review. From this activity, students reported diverse insights into the processes of data presentation, peer review, and scientific publishing. Analysis of preprint articles is therefore a valuable new tool to strengthen students' information literacy and understanding of the process of science.